docs: add roadmap page outlining future plans - #4149

Would maybe note the zarrs example here

good point, will add something crediting zarrs with this layout

I'm really conflicted on this, because I think baking fragmentation into the ecosystem is not great, but the flipside is that languages may have different features/advantages. It would be great to have a concrete reason (or list of reasons) as to why we would design around this level of extensibility

good points, I'd say we want the backend to be an extension point for a few reasons:

• we always want a traditional python impl. IMO it's our job as the python implementation, and it keeps the ecosystem healthy to have multiple implementations
• we can't guarantee uniform, complete feature support across all implementations. As long as we allow people to hack on stuff in python, there will be things the python impl does that e.g. a rust impl doesn't support, and that's OK. For now, a compiled backend is effectively an external dep, and we simply can't promise what external deps will do in perpetuity.
• IMO a full backend is just an refined version of our current model, which requires extenders manage buffer prototypes, data type implementations, codec implementations, and store implementations. Consolidating all of this pluggability on a single surface should make extending Zarr Python easier even for other python backends (like a cupy backend, or a pysparse backend)

@TomAugspurger it'd be great to get your thoughts on this component in particular.

FWIW I would like to prototype using a context manager in Zarrista for scope-local configuration.

See developmentseed/zarrista#153. In particular the idea is to borrow from rasterio.Env, which injects settings only for that local scope.

with rasterio.Env(GDAL_CACHEMAX=128000000) as env:
    # All drivers are registered, GDAL's raster block cache
    # size is set to 128 MB.
    # Commence processing...
    ...
    # End of processing.

# At this point, configuration options are set to their
# previous (possible unset) values.

This kind of thing relies on two routines or objects being the same, modulo the configuration. So why not make this explicit? for functions its easy, we declare their configuration in a parameter. for objects we can do something similar, by initializing the object with a configuration, and copying the object with a different configuration as needed, e.g. new_object = StatefulObject.with_config(new_config). IMO this is cleaner because it means runtime internals don't need to know about environment variables. they just inspect local data -- function parameters or the attributes of self -- for configuration info.

This kind of thing relies on two routines or objects being the same, modulo the configuration. So why not make this explicit?

We do currently make this explicit as a parameter into each function. The context manager would be an abstraction to automatically pass down the config into all zarrista calls within that scope.

I'm still not sure whether I like the idea though.

don't need to know about environment variables.

Oh but to be clear, I didn't have in mind environment variables. Despite the name of rasterio.Env, I don't think it's setting global environment variables, because that wouldn't be scope-local.

I had in mind a context manager that would hold config as a Python object, and that local config could be queried. But I don't think Zarrista should touch shell environment variables at all

that local config could be queried.

this is a pattern I don't really like. if an object like a function or class instance depends on some other data, then that should be formalized as part of the function signature or class structure. zarr-python today has a lot of places where code reaches for a free-floating config object, when IMO the simpler solution is to add parameters to functions / classes.

I'd like this section to be explicitly motivated for reasons beyond just cleaning up the shape of the internals.

this suggestion was based off an earlier version of the PR; I didn't realize it'd still be included in my review. It probably isn't still necessary, but I'll leave it.

I prefer keeping it as just "conventions" since specification may confuse folks who think of the core zarr spec